Determining image forensics using an estimated camera response function

ABSTRACT

An image forensics system estimates a camera response function (CRF) associated with a digital image, and compares the estimated CRF to a set of rules and compares the estimated CRF to a known CRF. The known CRF is associated with a make and a model of an image sensing device. The system applies a fusion analysis to results obtained from comparing the estimated CRF to a set of rules and from comparing the estimated CRF to the known CRF, and assesses the integrity of the digital image as a function of the fusion analysis.

RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S.application Ser. No. 15/198,810, filed on Jun. 30, 2016, the contents ofwhich are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to determining image forensics using anestimated camera response function.

BACKGROUND

Determining the integrity of digital media is of increasing importancedue to the proliferation of both real and forged imagery on social mediaplatforms. It is easier than ever to use manipulation programs (e.g.,Photoshop) to alter the content of an image in order to misinform thepublic or to commit fraud. As such, there is a need for methods toassess the integrity of imagery in both the commercial and governmentsectors. These methods must work with uncontrolled source imagery andproduce, with as little user input as possible, a numerical assessmentof the probability that the image or video has been altered in such away as to misinform or mislead the recipient.

In the government sector, DARPA has launched a program called MediFor(Media Forensics) to assess the integrity of visual media used byintelligence analysts for enemy force assessment, counter-intelligence,and to debunk misinformation from foreign intelligence services.

In the commercial realm, the ubiquity of digital cameras in mobilephones and other devices has made the assessment of integrityincreasingly important. Insurance adjusters have traditionally been auser of prior art digital image integrity assessment in order to assesswhether images of a car before or after an accident have been altered toexaggerate or understate damage. Other industries such as the parceldelivery industry have exhibited an interest in the use of digitalimagery to document the condition of a parcel when it comes into theircustody, and have similar concerns about the veracity ofcustomer-provided imagery.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are a block diagram illustrating features and operationsof a method and system for determining image forensics using anestimated camera response function.

FIG. 2 illustrates an example of a camera response function (CRF).

FIG. 3 illustrates an example of a camera response function thatincludes anomalies such as mid-function peaks and negative slopes.

FIG. 4 is a block diagram illustrating a system including a digitalcamera, a computer processor, and a database.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings that form a part hereof, and in which is shown by way ofillustration specific embodiments which may be practiced. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice the invention, and it is to be understood thatother embodiments may be utilized and that structural, electrical, andoptical changes may be made without departing from the scope of thepresent invention. The following description of example embodiments is,therefore, not to be taken in a limited sense, and the scope of thepresent invention is defined by the appended claims.

In an embodiment, a method of image forensics uses blur to estimatecertain camera parameters, and checks those parameters for consistency,at least, with a set of rules (both manually encoded and empiricallydetermined) which can detect manipulations without the need to accessimage metadata (JPEG or EXIF data, for example). In addition, whenmetadata is present and/or inferred using other means, more exactingchecks can be applied to detect whether the parameters are consistentwith the purported make and model of the camera.

The specific parameter estimated and checked in this method is thecamera response function (CRF), also known as the tone-mapping function.The CRF is a non-linear mapping from the photosensor's output to anintensity value (often an 8-bit value) used in the corresponding pixelof the resulting image. CRFs are used to improve the aesthetics ofphotographic imagery, since the raw photosensor response results inunpleasantly low contrast and harsh transitions at the ends of thedynamic range. Because the CRF's main goal is aesthetic, there is noobjectively best value and thus each manufacturer uses its ownproprietary CRF. To the extent that different camera models targetdifferent customer segments, CRFs exhibit variation even between modelsproduced by the same manufacturer. The role of CRFs with respect to blurhas only recently become well-known in the image processing and computervision literature.

Whereas blur has been used previously for image forensics, the prior artdepends on an unacceptably high degree of user input and performsrelative comparisons which increase the error of the resulting estimate.By contrast, methods to estimate the CRF from a natural (i.e., notcontrived) image exist, and the estimated CRF can be compared to both aset of general rules and a database of known CRFs. One such databasealready exists, but part of the present invention is the idea of aliving database updated by occasionally downloading and analyzingimagery from social media sites such as Flickr, Pintrest, etc.

In an embodiment, assessing the forensic integrity of an image or videobegins by estimating the CRF. This can be done in several ways. Thefirst, which is based on prior art, is to estimate the CRF via blurrededges in the image. An alternate method is to use image statistics, e.g.by computing the magnitudes of image gradients and modeling theirdistribution in relation to an a priori model of natural image gradientmagnitudes.

Once the CRF is estimated, several analyses are applied. First, theestimated CRF is compared to some rules from domain knowledge, e.g. thatthe CRF should be monotonic and that it should span the entire outputrange of the available bit depth. Second, the CRF is compared to adatabase of known CRFs, and features derived from both the current anddatabase CRFs are compared using anomaly detection methods to determinewhether the estimated CRF is likely to have come from the same space.Third, in the event that metadata are available purporting to documentthe camera's make and model, the estimated CRF is compared to thecorresponding entry from the database to check for consistency. Theoutputs of these three modules are then combined via a fusion approachfor an overall integrity assessment.

The maintenance of a CRF database can be important to the performance ofthe image forensics system. In an embodiment, in light of the changes tothe CRF space as new cameras are released, a CRF database is maintainedvia periodic analysis of imagery uploaded to social media sites. Forexample, Flickr has an application program interface (API) that allowsthis functionality of downloading a set of photos. In a first step, aset of recently uploaded images would be downloaded and thecorresponding metadata would be checked to make a list of current cameramakes and models. For any make or model where there is not a CRF in thedatabase, the system would download a second set of photos taken withthat particular camera. This second set of photos would then beprocessed to estimate a representative CRF for the camera make andmodel, and it would be added to the database.

FIGS. 1A and 1B are a block diagram illustrating features and operationsof a method for determining image forensics using an estimated cameraresponse function. FIGS. 1A and 1B include a number of process blocks110-168. Though arranged somewhat serially in the example of FIGS. 1Aand 1B, other examples may reorder the blocks, omit one or more blocks,and/or execute two or more blocks in parallel using multiple processorsor a single processor organized as two or more virtual machines orsub-processors. Moreover, still other examples can implement the blocksas one or more specific interconnected hardware or integrated circuitmodules with related control and data signals communicated between andthrough the modules. Thus, any process flow is applicable to software,firmware, hardware, and hybrid implementations.

Referring specifically to FIGS. 1A and 1B, at 110, a camera responsefunction (CRF) that is associated with a particular digital image isestimated. As indicated at 112, the CRF can be estimated using a blurrededge in the digital image. Specifically, the change in image intensityacross a motion-blurred edge would be linear in the absence of the CRF.The observed non-linearity of the blurred edge in the digital image canthus be used with a numerical optimization method to estimate the CRF.In under-constrained situations, prior information about the blur or theCRF shape can be imposed during optimization. In another embodiment, asindicated beginning at 114, the CRF is estimated by first computingimage statistics for the digital image. Pertinent image statistics canbe divided into spatial and frequency domain statistics. Spatial domainstatistics, for example, include histograms of image gradient magnitudeswhich, in the absence of blur and the CRF, have a known distribution; aswith edge methods, numerical optimization can be used to determine theCRF which best explains the difference between the observed and expectedimage statistics. Fourier domain statistics, for example, include theradial power spectrum of the image which, in the absence of blur and theCRF, have an expected falloff with the magnitude of the spatialfrequency; as before, numerical optimization can estimate the CRF whichbest explains the difference between the observed and expected powerspectra.

After the estimation of the CRF, at 120, the estimated CRF is comparedto a set of rules. As indicated at 122, the set of rules can include arule that the CRF is monotonic, a rule that the CRF spans an entireoutput range of an available bit depth, a rule that the CRF does notinclude mid-function peaks along the CRF, a rule that the CRF includesno negative slopes along the CRF, and a rule that the CRF for each colorchannel of the image sensing device has a substantially similar shape.To illustrate, FIG. 2 illustrates a typical CRF associated with adigital image that has not been tampered with. In contrast, FIG. 3illustrates a CRF in which the pixels in the image have been altered. Ascan be seen in FIG. 2, a normal CRF comprises a relatively smoothpositively increasing slope along its length. As illustrated in FIG. 3at 310, an altered digital image may cause the CRF to have minor peaksalong the length of the CRF, and also cause the CRF to have negativeslopes along the length of the CRF (320).

At 130, the estimated CRF is compared to a known CRF. The known CRF isassociated with a particular make and a model of a particular imagesensing device. The estimated and known CRFs can be compared indifferent ways. First, they can be compared directly, for instance bymeasuring the difference between them at corresponding points andcomputing a root mean squared difference. Alternatively, CRFs can becompared via features. More specifically, as outlined at 132, a featurefrom the estimated CRF is compared to a feature from the known CRF. Suchfeatures for comparison can include a histogram of slope values alongthe CRF, a measured area under the curve, and the start- and end-pointsof the middle linear section. Then, at 134, an anomaly detection is usedto determine whether the estimated CRF is likely to have come from thesame image sensing device. As a practical matter, large sets of CRFs areused in anomaly detection so as to increase the accuracy of thedetection process.

As indicated at 136, the known CRF is updated via an analysis of animage from a website. More specifically, this updating of the known CRFincludes downloading an image from a website (136A), receiving metadatafrom a header in the downloaded image (136B), determining, from themetadata from the header in the downloaded image a make and model of animage sensing device that created the downloaded image (136C), andstoring the metadata, make, and model of the image sensing device thatcreated the downloaded image in the database (136D). In a furtherembodiment, at 136E, it is determined if the make and model of the imagesensing device that created the downloaded image are currently in thedatabase. If not, at 136F, a second image is downloaded that wasgenerated using the make and the model of the image sensing device thatcreated the downloaded image. Then, at 136G, the second image is used toestimate a CRF for the make and model of the image sensing device thatcreated the downloaded image, and at 136H, the CRF for the make andmodel of the image sensing device that created the downloaded image isstored in the database.

At 140, a fusion analysis is applied to the results obtained fromcomparing the estimated CRF to a set of rules and from comparing theestimated CRF to the known CRF. In an embodiment, a fusion analysis caninvolve one or more of the following methods. A decision-level fusion,for example voting, where different cues are used to independentlypredict something (such as that an image has been forged), and thosebinary decisions are combined. A common option is to take a majorityvote by the independent predictions. A score-level fusion, where each ofthe independent features is used to predict a continuous score(conceptually, the probability that the image was forged) and thosecontinuous scores are fused to generate a single system output, which isa binary indicator of whether the image was forged. A feature-levelfusion, where lower-level features (for example, something like thederivatives of the CRF in different places) are combined (for example,by concatenating them), and a single machine learning method is appliedto the concatenated feature vector to predict whether the image wasforged. Thereafter, at 150, the integrity of the image is assessed as afunction of the fusion analysis.

In yet another embodiment, as indicated at 160, metadata that is storedin a header of the digital image is received into a processing unit. Asis the case with typical header data, the metadata stored in the headerof the digital image identifies a make and a model of an image sensingdevice that created the digital image. At 162, metadata that are storedin a database are then received into the processing unit. The metadatastored in the database are associated with the make and model of theimage sensing device that created the digital image. At 164, themetadata stored in the header of the digital image is compared to themetadata stored in the database that is associated with the make andmodel of the image sensing device that created the digital image. At166, the fusion analysis is applied to the results from comparing themetadata stored in the header of the digital image to the metadatastored in the database that are associated with the make and model ofthe image sensing device that created the digital image, and at 168, theintegrity of the image is assessed as a function of the fusion analysis.

FIG. 4 is a block diagram illustrating a system 400 including a digitalcamera 420, a computer processor 410, and a database 430. The system 400can be used to execute the several functions outlined in FIGS. 1A and1B.

It should be understood that there exist implementations of othervariations and modifications of the invention and its various aspects,as may be readily apparent, for example, to those of ordinary skill inthe art, and that the invention is not limited by specific embodimentsdescribed herein. Features and embodiments described above may becombined with each other in different combinations. It is thereforecontemplated to cover any and all modifications, variations,combinations or equivalents that fall within the scope of the presentinvention.

The Abstract is provided to comply with 37 C.F.R. § 1.72(b) and willallow the reader to quickly ascertain the nature and gist of thetechnical disclosure. It is submitted with the understanding that itwill not be used to interpret or limit the scope or meaning of theclaims.

In the foregoing description of the embodiments, various features aregrouped together in a single embodiment for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting that the claimed embodiments have more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle disclosed embodiment. Thus the following claims are herebyincorporated into the Description of the Embodiments, with each claimstanding on its own as a separate example embodiment.

The invention claimed is:
 1. A process to assess integrity in a digitalimage comprising: estimating a camera response function (CRF) associatedwith the digital image by computing a plurality of magnitudes of imagegradients and modeling distribution in relation to a priori model ofnatural image gradient magnitudes; comparing the estimated CRF to a setof rules; receiving metadata stored in a header of the digital image,the metadata stored in the header of the digital image identifying amake and a model of an image sensing device that created the digitalimage; receiving metadata stored in a database, the metadata stored inthe database associated with the make and model of the image sensingdevice that created the digital image; comparing the metadata stored inthe header of the digital image to the metadata stored in the databasethat is associated with the make and model of the image sensing devicethat created the digital image; applying a fusion analysis to resultsobtained from comparing the estimated CRF to a set of rules and fromcomparing the metadata stored in the header of the digital image to themetadata stored in the database that is associated with the make andmodel of the image sensing device that created the digital image; andassessing the integrity of the image as a function of the fusionanalysis.
 2. The process of claim 1, wherein the estimated CRF isestimated using a blurred edge in the digital image.
 3. The process ofclaim 1, comprising computing image statistics for the digital image,the image statistics to be used with optimization methods to infer theCRF.
 4. The process of claim 3, wherein the image statistics arecomputed in the spatial domain.
 5. The process of claim 3, wherein theimage statistics are computed in a Fourier domain.
 6. The process ofclaim 1, wherein the set of rules comprises a rule that the CRF ismonotonic, a rule that the CRF spans an entire output range of anavailable bit depth, that the CRF comprises no mid-function peaks alongthe CRF, that the CRF comprises no negative slopes, that the CRFcomprises no mid-function zero slopes, and that a CRF for each colorchannel of the image sensing device has a substantially similar shape.7. The process of claim 1, wherein the fusion analysis comprises one ormore of a decision-level fusion, a score-level fusion, and afeature-level fusion.
 8. A process to assess integrity in a digitalimage comprising: estimating a camera response function (CRF) associatedwith the digital image by computing a plurality of magnitudes of imagegradients and modeling distribution in relation to a priori model ofnatural image gradient magnitudes; comparing the estimated CRF to a setof rules; applying a fusion analysis to results obtained from comparingthe estimated CRF to a set of rules; and assessing the integrity of thedigital image as a function of the fusion analysis.
 9. The process ofclaim 8, comprising: receiving metadata stored in a header of thedigital image, the metadata stored in the header of the digital imageidentifying a make and a model of an image sensing device that createdthe digital image; receiving metadata stored in a database, the metadatastored in the database associated with the make and model of the imagesensing device that created the digital image; comparing the metadatastored in the header of the digital image to the metadata stored in thedatabase that is associated with the make and model of the image sensingdevice that created the digital image; applying the fusion analysis toresults obtained from comparing the metadata stored in the header of thedigital image to the metadata stored in the database that is associatedwith the make and model of the image sensing device that created thedigital image; and assessing the integrity of the digital image as afunction of the fusion analysis.
 10. The process of claim 8, wherein theestimated CRF is estimated using a blurred edge in the digital image.11. The process of claim 8, comprising computing image statistics forthe digital image, the image statistics to be used with optimizationmethods to infer the CRF; wherein the image statistics are computed inthe spatial domain or the image statistics are computed in the Fourierdomain.
 12. The process of claim 8, wherein the set of rules comprises arule that the CRF is monotonic, a rule that the CRF spans an entireoutput range of an available bit depth, that the CRF comprises nomid-function peaks along the CRF, that the CRF comprises no negativeslopes, that the CRF comprises no mid-function zero slopes, and that theCRF for each color channel of the image sensing device has asubstantially similar shape; and wherein the fusion analysis comprisesone or more of a decision-level fusion, a score-level fusion, and afeature-level fusion.
 13. An image forensics system comprising: one ormore processors; and a computer readable medium storing instructionsthat, when executed by the one or more processors, cause the system toperform operations comprising: estimating a camera response function(CRF) associated with a digital image by computing a plurality ofmagnitudes of image gradients and modeling distribution in relation to apriori model of natural image gradient magnitudes; comparing theestimated CRF to a set of rules; applying a fusion analysis to resultsobtained from comparing the estimated CRF to a set of rules; andassessing the integrity of the digital image as a function of the fusionanalysis.
 14. The image forensics system of claim 13, comprising:receiving metadata stored in a header of the digital image, the metadatastored in the header of the digital image identifying a make and a modelof an image sensing device that created the digital image; receivingmetadata stored in a database, the metadata stored in the databaseassociated with the make and model of the image sensing device thatcreated the digital image; comparing the metadata stored in the headerof the digital image to the metadata stored in the database that isassociated with the make and model of the image sensing device thatcreated the digital image; applying the fusion analysis to resultsobtain from comparing the metadata stored in the header of the digitalimage to the metadata stored in the database that is associated with themake and model of the image sensing device that created the digitalimage; and assessing the integrity of the digital image as a function ofthe fusion analysis.
 15. The image forensics system of claim 13, whereinthe estimated CRF is estimated using a blurred edge in the digitalimage.
 16. The image forensics system of claim 13, comprising computingimage statistics for the digital image, the image statistics beingcomputed in the spatial or Fourier domain; and wherein the estimated CRFis estimated by numerical optimization to explain the difference betweenobserved and expected statistics.
 17. The image forensics system ofclaim 13, wherein the set of rules comprises a rule that the CRF ismonotonic, a rule that the CRF spans an entire output range of anavailable bit depth, that the CRF comprises no mid-range peaks along theCRF, that the CRF comprises no negative slopes, that the CRF comprisesno mid-function zero slopes, and that a CRF for each color channel ofthe image sensing device has a substantially similar shape.
 18. Theimage forensics system of claim 13, comprising instructions forcomputing image statistics for the digital image, the image statisticsto be used with optimization methods to infer the CRF.
 19. The imageforensics system of claim 18, wherein the image statistics are computedin the spatial domain or the image statistics are computed in a Fourierdomain.
 20. The image forensics system of claim 13, wherein the fusionanalysis comprises one or more of a decision-level fusion, a score-levelfusion, and a feature-level fusion.